refactor: use high watermark to finish backfill faster #18342
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
9 tasks
This stack of pull requests is managed by Graphite. Learn more about stacking. |
9 tasks
xxchan
force-pushed
the
08-24-refactor_use_high_watermark_to_finish_backfill_faster
branch
from
6fb622b to
0eaf49f
Compare
xxchan
requested review from
chenzl25,
fuyufjh,
tabVersion,
BugenZhao and
st1page
xxchan
force-pushed
the
08-24-refactor_add_target_offsets_to_determinine_if_source_backfill_finished
branch
from
aa5610d to
272a7a4
Compare
xxchan
force-pushed
the
08-24-refactor_use_high_watermark_to_finish_backfill_faster
branch
from
0eaf49f to
d2bc71a
Compare
9 tasks
xxchan
force-pushed
the
08-24-refactor_add_target_offsets_to_determinine_if_source_backfill_finished
branch
from
c8b80b7 to
2b8f062
Compare
xxchan
force-pushed
the
08-24-refactor_use_high_watermark_to_finish_backfill_faster
branch
from
d2bc71a to
051239b
Compare
9 tasks
Base automatically changed from
08-24-refactor_add_target_offsets_to_determinine_if_source_backfill_finished
to
main
September 3, 2024 04:05
xxchan
force-pushed
the
08-24-refactor_use_high_watermark_to_finish_backfill_faster
branch
from
efc2a77 to
dc3aae9
Compare
9 tasks
xxchan
force-pushed
the
08-24-refactor_use_high_watermark_to_finish_backfill_faster
branch
from
4a5c008 to
1a663a9
Compare
chenzl25
approved these changes
Sep 3, 2024
| @@ -21,7 +44,7 @@ create source s0 (v1 int, v2 varchar) with ( | |||
| scan.startup.mode = 'earliest' | |||
| ) FORMAT PLAIN ENCODE JSON; | |||
|
|
|||
| query I | |||
| query ? | |||
There was a problem hiding this comment.
When do we need to use ? instead of I ?
There was a problem hiding this comment.
currently sqllogictest --override will produce ?. Actually the character doesn't have any meaning now, any character will pass test..
| system ok | ||
| internal_table.mjs --name s0 --type source | ||
| ---- | ||
| (empty) | ||
|
|
||
|
|
||
| # offset 0 must be backfilled, not from upstream. | ||
| # SourceBackfill starts from offset 0, with backfill_info: HasDataToBackfill { latest_offset: "0" } (decided by kafka high watermark). |
There was a problem hiding this comment.
In this case, what will the high watermark value be?
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
10 tasks
xxchan
deleted the
08-24-refactor_use_high_watermark_to_finish_backfill_faster
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
step 2 & 3 of #18299, see there for the motivation and a state diagram of the new design.
Checklist
./risedev check(or alias,./risedev c)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.